Systems and methods for dynamic demand sensing and forecast adjustment

ABSTRACT

System and method relating to demand forecasting and readjusting forecasts based on forecast error.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a US continuation of U.S. patent applicationSer. No. 18/456,111 filed Aug. 25, 2023, which is a U.S.continuation-in-part patent application of U.S. patent application Ser.No. 18/071,802 filed Nov. 30, 2022, which is a continuation applicationof U.S. patent Ser. No. 11,537,825 filed Apr. 1, 2020, which is acontinuation-in-part of U.S. patent Ser. No. 11,526,899 filed on Oct.11, 2019, and the present application claims priority under 35 U.S.C.Section 120, or otherwise, to U.S. provisional patent application Ser.No. 63/518,713, filed Aug. 10, 2023, all of which are incorporatedherein by reference.

BACKGROUND

Demand sensing uses artificial intelligence and machine learningalgorithms to capture short-term demand patterns. Demand sensing is avaluable technique in the marketplace for forecasting, however, once ademand sensing machine learning pipeline is in production, changes tothe pipeline are difficult and costly to implement.

BRIEF SUMMARY

In one aspect, a computer-implemented method includes using a firstforecast model for providing first forecast data for a first forecastwindow, the first forecast data for one or more store locations,transmitting the first forecast data to a user, receiving first salesdata for the one or more store locations for a second time interval, thesecond time interval subsequent a first time interval, determining anerror in the first forecast data based on the first forecast data andthe first sales data, removing the error from the first forecast datafor forming first adjusted forecast data, the first adjusted forecastdata for the one or more store locations, and transmitting the firstadjusted forecast data to the user.

The method may also include collecting historical data for the firsttime interval, generating first feature data based on the historicaldata, and processing the first feature data by a processor for traininga machine learning algorithm for forming the first forecast model.

The method may also include where determining the error in the firstforecast data based on the first forecast data and the first sales dataincludes determining the error between the first forecast data for eachof the one or more store locations and the first sales data for each ofthe one or more store locations corresponding thereto.

The method may also include where determining the error in the firstforecast data based on the first forecast data and the first sales dataincludes determining a percentage error between the first forecast datafor each of the one or more store locations and the first sales data foreach of the one or more store locations and an average percentage errorthereof.

The method may also further include collecting second data for a thirdtime interval, the second data including at least historical data andthe first sales data, generating second feature data based on the seconddata, processing second feature data by a processor for training amachine learning algorithm for forming a second forecast model, usingthe second forecast model for providing second forecast data for asecond forecast window, the second forecast data including secondforecast data for the one or more store locations, and transmitting thesecond forecast data to the user.

The method may also further include receiving second sales data, thesecond sales data for the one or more store locations during a fourthtime interval subsequent the second time interval, determining an errorin the first forecast data based on the first forecast data and thesecond sales data, removing an error from the first forecast data forforming third forecast data, and transmitting the third forecast data tothe user.

The method may also include where removing the error from the firstforecast data for forming the first adjusted forecast data includesremoving the average percentage error from the first forecast data foreach of the one or more store locations, the first adjusted forecastdata includes first adjusted forecast data for each of the one or morestore locations.

In one aspect, a non-transitory computer-readable storage medium, thecomputer-readable storage medium including instructions that whenexecuted by a computer, cause the computer to use a first forecast modelfor providing first forecast data for a first forecast window, the firstforecast data for one or more store locations, transmit the firstforecast data to a user, receive first sales data for the one or morestore locations for a second time interval, the second time intervalsubsequent a first time interval, determine an error in the firstforecast data based on the first forecast data and the first sales data,remove the error from the first forecast data for forming first adjustedforecast data, the first adjusted forecast data for the one or morestore locations, and transmit the first adjusted forecast data to theuser.

The non-transitory computer-readable storage medium, where thecomputer-readable storage medium further includes instructions that whenexecuted by a computer, cause the computer to collect historical datafor the first time interval, generate first feature data based on thehistorical data, and process the first feature data by a processor fortraining a machine learning algorithm for forming the first forecastmodel.

The non-transitory computer-readable storage medium may also includewhere determining the error in the first forecast data based on thefirst forecast data and the first sales data includes determining theerror between the first forecast data for each of the one or more storelocations and the first sales data for each of the one or more storelocations corresponding thereto.

The non-transitory computer-readable storage medium may also includewhere determining the error in the first forecast data based on thefirst forecast data and the first sales data includes determining apercentage error between the first forecast data for each of the one ormore store locations and the first sales data for each of the one ormore store locations and an average percentage error thereof.

The non-transitory computer-readable storage medium may also includewhere removing the error from the first forecast data for forming thefirst adjusted forecast data includes removing the average percentageerror from the first forecast data for each of the one or more storelocations, the first adjusted forecast data includes first adjustedforecast data for each of the one or more store locations.

The non-transitory computer-readable storage medium, where thecomputer-readable storage medium may also include instructions that whenexecuted by a computer, cause the computer to collect second data for athird time interval, the second data including at least historical dataand the first sales data, generate second feature data based on thesecond data, process second feature data by the processor for training amachine learning algorithm for forming a second forecast model, use thesecond forecast model for providing second forecast data for a secondforecast window, the second forecast data include second forecast datafor the one or more store locations, and transmit the second forecastdata to the user.

The non-transitory computer-readable storage medium, where thecomputer-readable storage medium may also include instructions that whenexecuted by a computer, cause the computer to receive second sales data,the second sales data for the one or more store locations during afourth time interval subsequent the second time interval, determine anerror in the first forecast data based on the first forecast data andthe second sales data, remove an error from the first forecast data forforming third forecast data, and transmit the third forecast data to theuser.

In one aspect, a computing apparatus includes a processor. The computingapparatus also includes a memory storing instructions that, whenexecuted by the processor, configures the apparatus to use a firstforecast model for providing first forecast data for a first forecastwindow, the first forecast data for one or more store locations,transmit the first forecast data to a user, receive first sales data forthe one or more store locations for a second time interval, the secondtime interval subsequent a first time interval, determine an error inthe first forecast data based on the first forecast data and the firstsales data, remove the error from the first forecast data for formingfirst adjusted forecast data, the first adjusted forecast data for theone or more store locations, and transmit the first adjusted forecastdata to the user.

The computing apparatus may also be configured to collect historicaldata for the first time interval, generate first feature data based onthe historical data, and process the first feature data by the processorfor training a machine learning algorithm for forming the first forecastmodel.

The computing apparatus may also be configured where determining theerror in the first forecast data based on the first forecast data andthe first sales data includes determining the error between the firstforecast data for each of the one or more store locations and the firstsales data for each of the one or more store locations correspondingthereto.

The computing apparatus may also be configured where determining theerror in the first forecast data based on the first forecast data andthe first sales data includes determining a percentage error between thefirst forecast data for each of the one or more store locations and thefirst sales data for each of the one or more store locations and anaverage percentage error thereof.

The computing apparatus may also be configured where removing the errorfrom the first forecast data for forming the first adjusted forecastdata includes removing the average percentage error from the firstforecast data for each of the one or more store locations, the firstadjusted forecast data includes first adjusted forecast data for each ofthe one or more store locations.

The computing apparatus may also be further configured to collect seconddata for a third time interval, the second data including at leasthistorical data and the first sales data, generate second feature databased on the second data, process second feature data by the processorfor training a machine learning algorithm for forming a second forecastmodel, use the second forecast model for providing second forecast datafor a second forecast window, the second forecast data include secondforecast data for the one or more store locations, and transmit thesecond forecast data to the user.

The computing apparatus may also be further configured to receive secondsales data, the second sales data for the one or more store locationsduring a fourth time interval subsequent the second time interval,determine an error in the first forecast data based on the firstforecast data and the second sales data, remove an error from the firstforecast data for forming third forecast data, and transmit the thirdforecast data to the user.

The details of one or more embodiments of the subject matter of thisspecification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

To easily identify the discussion of any particular element or act, themost significant digit or digits in a reference number refer to thefigure number in which that element is first introduced.

FIG. 1 illustrates a system architecture in accordance with oneembodiment.

FIG. 2 illustrates a block diagram in accordance with one embodiment.

FIG. 3 illustrates a flowchart for an ingestion module in accordancewith one embodiment.

FIG. 4 illustrates a transformation examples in accordance with oneembodiment.

FIG. 5 illustrates a transformation examples in accordance with oneembodiment.

FIG. 6 illustrates a flowchart for an external data module in accordancewith one embodiment.

FIG. 7 illustrates a transformation examples in accordance with oneembodiment.

FIG. 8 illustrates a flowchart for a monitor module in accordance withone embodiment.

FIG. 9 illustrates a flowchart for a forecasting module in accordancewith one embodiment.

FIG. 10 illustrates a flowchart of a forecast request evaluation by amonitor module in accordance with one embodiment.

FIG. 11 illustrates a flow chart of a forecasting module in accordancewith one embodiment.

FIG. 12 illustrates a flow chart of a forecasting module in accordancewith one embodiment.

FIG. 13 illustrates a machine learning forecast in accordance with oneembodiment.

FIG. 14 illustrates a simple flow chart of a method for forecastingusing a machine learning technique in accordance with an embodiment.

FIG. 15 illustrates a simple block diagram of a system in accordancewith an embodiment.

FIG. 16 illustrates is simple flow chart of a method for adjusting aforecast provided by a forecast model in accordance with an embodiment.

FIG. 17 illustrates exemplary values of forecast data and adjustedforecast data in accordance with an embodiment.

FIG. 18 illustrates a graphical representation of forecast data andadjusted forecast data in accordance with an embodiment.

DETAILED DESCRIPTION

The demand sensing method can provide predicted daily sales for a singleproducts (for example, according to their stock keeping unit (SKU)identification codes) for single locations (e.g. retail stores) oversome horizon (e.g. 13 weeks ahead) for a variety of purposes, including:allowance by the user to use the predictions to drive replenishmentorders at the defined locations; and gaining an analytical understandingof the factors driving the predicted sales in order to plan for thefuture.

The data processing services are composed of various components of amachine learning pipeline. Per user request, features may be generatedfrom the raw user-specific and public datasets. Then one or morequantile regression models can be trained with these features. Selectionof features and hyperparameters can be achieved through the evaluationof each model on the same validation set. The evaluation comprisesmanaging a simulated inventory for the period of time equivalent to thevalidation set, where orders are given based on simple heuristics andkey performance metrics are measured, such as excessive inventory over aperiod of time and number of stock out days. Once a model is chosen (forbest performance for an item and store combination), the contribution ofeach feature (on the demand predictions) may be evaluated through modelinterpretation techniques (e.g. SHapley Additive exPlantions). In a laststep, data related to predictions, prediction quality, and predictioncontributions may be gathered and illustrated to the user by a number ofinteractive visualizations that are found in user-application interfacesmentioned above.

Historical data may be collected from a variety of sources. For example,data may be collected from a client/user that includes historical plusforwarding looking data such as campaigns. In some embodiments,historical client data can include point-of-sales data that providesinformation on the amount of product sold at a particular day at aparticular location; and inventory of a particular product at aparticular location. Other types of data can be mined from the web andsocial media, such as weather data, financial markets, and the like.Calendar data that includes local holidays, along with local event datamay also be collected. Promotion campaign details for a particularproduct at a particular location can also be included, and otherrelevant events. In summary, any information that relates to, or impactsupon, the sales of a particular product at a particular location, can beused as part of the input dataset.

The raw data is first processed, before being used by machine learningmodels. In some embodiments, that can entail features generation (whichis fed into the various models of the Machine Learning Module).

In some embodiments, the data is partitioned into a training portion anda validation portion. Each model may be trained on the training portion.Each trained model is then tested on the remaining validation portion ofthe data, and the model that provides the best prediction on thevalidation portion of the data, is selected. The selected machinelearning model can be retrained on the entire data set, and then usedfor deployment on live data.

In some embodiments, the data is partitioned into a training portion, avalidation portion and a testing portion. Each model may be trained onthe training portion. Each trained model is then tested on thevalidation portion of the data, and the model that provides the bestprediction on the validation portion of the data, is selected. Theselected machine learning model can be retrained on the first twoportions (i.e. data from both the training and validation portions). Itcan then be tested on the testing portion to provide a measure ofaccuracy for the user. The selected model may then be used fordeployment on live data.

The machine learning models are trained and validated on data pertainingto a particular product at a particular location, in order to provideforecasting for that product at that location. Forecasting by theselected trained model can be for a selected time frame. In someembodiments, that time frame is a 13-week horizon. The “live” input dataused for forecasting can include sales data from a previous time period(e.g. sales from 1 month, or 2 months, or 3 months ago, or more);promotion campaigns, weather data for the location and in the vicinityof the location, market indexes for the location and in the vicinity ofthe location; and events at or in the vicinity of the location.

FIG. 1 illustrates a system architecture 100 in accordance with oneembodiment.

Client data source 102 provides a variety of raw data from the user,including, but not limited to: point of sales data that indicates thesales record of all of the client's products at every location; theinventory history of all of the client's products at every location;promotional campaign details for all products at all locations, andevents that are important/relevant for sales of a client's product atevery location.

Ingestion module 302 receives client data source 102 and converts itinto a different format that is ready for storage in ML storage 106.

Dynamic demand sensing module 120 comprises three modules: external datamodule 110; monitor module 112 and forecasting module 114, each of whichis described below.

External data module 110 obtains external data from external datasource(s) 108 and prepares it for storage in ML storage 106. Externaldata can be mined from the web and social media, and can includeinformation such as weather data, financial market data, and the like.for a plurality of locations. These locations can include locations thatare important to the client's product sales. Furthermore, marketindicines may be wider than the actual location(s) required by theclient, as market indices are often nation-wide, whereas locations canbe confined to smaller geographical areas such as cities or towns.Calendar data that includes local holidays, along with local event datacan also be included. In some embodiments, external data module 110 canbe on a schedule to obtain external data from external data source(s)108 (e.g. on a daily, semi-daily, weekly, bi-weekly, monthly, etc.schedule). External data module 110 can access different types of data,each on its own schedule. For example, weather data can be accessed on adaily basis; market data can be accessed daily or weekly, calendar datacan be accessed yearly, etc.

Request module 116 provides a request to the monitor module 112, toprovide a forecast for a subset of the data stored in ML storage 106. Insome embodiments, request module 116 may request the forecasting of aparticular product at a particular store (or location) for a forecastinghorizon time period.

Monitor module 112 obtains information provided by request module 116 asto which subset of products/stores are to be forecasted. Based oninformation related to the subset and information related to the MLstorage 106, the monitor module 112 decides on how the subset will beforecast, based on a set of rules, described below. Once the subset isready for forecasting, forecasting module 114 provides a forecast thatis output to output 118.

FIG. 2 illustrates a system 200 in accordance with one embodiment of adynamic demand sensing system.

System 200 includes a system server 202, ML storage 106, client datasource 102 and external data source(s) 108. System server 202 caninclude a memory 206, a disk 208, a processor 204 and a dynamic demandsensing module 120. While one processor 204 is shown, the system server202 can comprise one or more processors. In some embodiments, memory 206can be volatile memory, compared with disk 208 which can be non-volatilememory. In some embodiments, system server 202 can communicate with MLstorage 106, external data source(s) 108 and client data source 102 vianetwork 210. While ML storage 106 is illustrated as separate from systemserver 202, ML storage 106 can also be integrated into system server202, either as a separate component within system server 202, or as partof at least one of memory 206 and disk 208.

System 200 can also include additional features and/or functionality.For example, system 200 can also include additional storage (removableand/or non-removable) including, but not limited to, magnetic or opticaldisks or tape. Such additional storage is illustrated in FIG. 2 bymemory 206 and disk 208. Storage media can include volatile andnonvolatile, removable and non-removable media implemented in any methodor technology for storage of information such as computer-readableinstructions, data structures, program modules or other data. Memory 206and disk 208 are examples of non-transitory computer-readable storagemedia. Non-transitory computer-readable media also includes, but is notlimited to, Random Access Memory (RAM), Read-Only Memory (ROM),Electrically Erasable Programmable Read-Only Memory (EEPROM), flashmemory and/or other memory technology, Compact Disc Read-Only Memory(CD-ROM), digital versatile discs (DVD), and/or other optical storage,magnetic cassettes, magnetic tape, magnetic disk storage or othermagnetic storage devices, and/or any other medium which can be used tostore the desired information and which can be accessed by system 200.Any such non-transitory computer-readable storage media can be part ofsystem 200. Communication between system server 202, client data source102, ML storage 106 and external data source(s) 108 via network 210 canbe over various network types. Non-limiting example network types caninclude Fibre Channel, small computer system interface (SCSI),Bluetooth, Ethernet, Wi-fi, Infrared Data Association (IrDA), Local areanetworks (LAN), Wireless Local area networks (WLAN), wide area networks(WAN) such as the Internet, serial, and universal serial bus (USB).Generally, communication between various components of system 200 maytake place over hard-wired, cellular, Wi-Fi or Bluetooth networkedcomponents or the like. In some embodiments, one or more electronicdevices of system 200 may include cloud-based features, such ascloud-based memory storage.

ML storage 106 may implement an “in-memory” database, in which volatile(e.g., non-disk-based) storage (e.g., Random Access Memory) is used bothfor cache memory and for storing the full database during operation, andpersistent storage (e.g., one or more fixed disks) is used for offlinepersistency and maintenance of database snapshots. Alternatively,volatile storage may be used as cache memory for storing recently-useddata, while persistent storage stores the full database.

ML storage 106 may store metadata regarding the structure, relationshipsand meaning of data. This information may include data defining theschema of database tables stored within the data. A database tableschema may specify the name of the database table, columns of thedatabase table, the data type associated with each column, and otherinformation associated with the database table. ML storage 106 may alsoor alternatively support multi-tenancy by providing multiple logicaldatabase systems which are programmatically isolated from one another.Moreover, the data may be indexed and/or selectively replicated in anindex to allow fast searching and retrieval thereof. In addition, MLstorage 106 can store a number of machine learning models that areaccessed by the dynamic demand sensing module 120. A number of MLmodels, such as gradient-boosted trees, ensemble of trees and supportvector regression, can be used.

External data source(s) 108 may include sources that provide bothhistorical and real-time information about climate, weather, financialindexes, web data, social media data, local holidays, festivals andevents, and the like. Commercial weather and financial services can bepurchased or accessed by other means.

Client data source 102 may provide a variety of raw data from the user,including, but not limited to: point of sales data that indicates thesales record of all of the client's products at every location; theinventory history of all of the client's products at every location;promotional campaign details for all products at all locations, andevents that are important/relevant for sales of a client's product atevery location.

Using network 210, system server 202 can retrieve data from ML storage106, 102 and external data source(s) 108. The retrieved data can besaved in memory 206 or disk 208. In some cases, system server 202 canalso comprise a web server, and can format resources into a formatsuitable to be displayed on a web browser.

FIG. 3 illustrates a flowchart 300 for an ingestion module 302 inaccordance with one embodiment.

Ingestion module 302 receives data from client data source 102 at block304, and proceeds to pre-process the data at block 306. Pre-processingmay include transformation, validation, remediation, or any combinationthereof, of the data.

Validation of the data simple means to determine whether there arepotential errors in the incoming data. For example, validation caninclude identification of missing data, null data, differences in rowcounts and data mismatches. In some embodiments, data validation modulemay use a machine learning algorithm in conjunction with a z-scorethreshold value to identify anomalous data values.

Data remediation involves remediation or re-calculation of data that isindicative of an error. For example: missing or erroneous values may bereplaced using data that is interpolated from an existing value orvalues, an average of existing data or a mean of existing data. In someembodiments, remediation of data can use a predictive model to replacedata that is indicative of error.

Transformation examples are illustrated in FIG. 4 and described below.

The pre-processed data is used to update data at block 308, for storagein the ML storage 106.

FIG. 4 illustrates a transformation examples 400 in accordance with oneembodiment. Examples of features 402 can include data related to: pointof sales, weather, events/holidays, market index, web traffic andpromotions. Features 402 may include additional categories of data,fewer, or different categories than those shown in FIG. 4 .

-   -   Example 1 404, shows how data related to a rare event, which is        in binary form, is transformed to a form that includes integers,        by specifying the number of days to the event. For example, the        rare event can have the value ‘0’ to indicate the day a store is        open (e.g. Mon-Sat) and ‘1’ to indicate the day a store is        closed (e.g. Sunday). The series of ‘0’s and ‘1’s is        transformed, instead, to a series of integers that indicate how        many days away that a given day is to the rare event.    -   Example 2 406 shows an example of transforming consecutive dates        to a tabular form that lists year (in one row); month (in a        second row) and date (in the third row).    -   Example 3 408 shows an example of transforming temperature        values on certain dates, to temperature values in relation to        the lowest temperature reading (6° C.). The original 6° C.        reading is transformed to ‘0’; 7° C. to ‘1’; 8° C. to ‘2’, and        so forth. Graphical representations of transformations are        discussed below.

FIG. 5 illustrates further transformation examples 500 in accordancewith one embodiment.

Promotion transformation 504 can be graphically represented as thenumber of days after a promotion. rather than the actual day of thepromotion. For comparison, sales of an item 502 is shown next to salespromotion transformation 504. Preliminary analysis shows that there wasa spike in sales shortly after a promotion in October 2016 and April2017.

FIG. 6 illustrates a flowchart 600 for an external data module 110 inaccordance with one embodiment.

External data module 110 fetches data (at block 602) from external datasource(s) 108 which can include raw data about weather, market indices,trends, etc. The external data source(s) 108 provide data thatcomplements client data source 102 (of FIG. 1 ). The raw data is cleaned(or validated) to remove outliers, and transformed (at block 604) forstorage, at block 606, in the ML storage 106.

Pre-processing may include transformation, validation, remediation, orany combination thereof, of the data.

Validation of the data simple means to determine whether there arepotential errors in the incoming data. For example, validation caninclude identification of missing data, null data, differences in rowcounts and data mismatches. In some embodiments, data validation modulemay use a machine learning algorithm in conjunction with a z-scorethreshold value to identify anomalous data values.

Data remediation involves remediation or re-calculation of data that isindicative of an error. For example: missing or erroneous values may bereplaced using data that is interpolated from an existing value orvalues, an average of existing data or a mean of existing data. In someembodiments, remediation of data can use a predictive model to replacedata that is indicative of error.

Examples of transformation of external data are discussed with referenceto FIG. 7 .

The pre-processed external data is used to update data at block 308, forstorage in the ML storage 106.

FIG. 7 illustrates further transformation examples 700 of external datain accordance with one embodiment.

Examples of external data include calendar dates, such as holidays, andweather data.

Holidays transformation 702 can be graphically represented as the numberof days until a holiday, rather than the actual day of the holiday. Forcomparison, sales of an item 704 is shown next to holidaystransformation 702. Preliminary analysis suggests that there was a spikein sales shortly after a holiday in October 2016 and April 2017.

Weather transformation 706 can be graphically represented as a weeklyrolling average of sky coverage over an area where the item is sold. Forcomparison, sales of an item 704 is shown next to holidaystransformation 702.

FIG. 8 illustrates a flowchart 800 for a monitor module 112 inaccordance with one embodiment.

Monitor module 112 receives a request for a forecast (at block 802) fromrequest module 116. Based on the specific information associated withthe request, monitor module 112 evaluates the forecasting step (at block804) based on the data and machine learning metrics, all stored in MLstorage 106. Details of this evaluation are described below. Once theforecasting step is evaluated, instructions are sent to the forecastingmodule 114 (at block 806).

FIG. 9 illustrates a flowchart 900 for a forecasting module inaccordance with one embodiment.

Monitor module 112 evaluates a forecast request from request module 116,in view of data and machine learning model metrics stored in ML storage106. Instructions are sent to the forecasting module 114 to initiate oneof three task sequences. A first task sequence comprises: forecastingmodule 114 will select a machine learning model at block 902, followedby training the selected model at block 904, followed by forecastingwith the trained model at block 906, for a specified future period oftime.

A second, different task sequence comprises: training a selected modelat block 904, followed by forecasting with the trained model at block906, for a specified future period of time.

A third, different task sequence simply comprises forecasting with aselected, trained model at block 906, for a specified future period oftime.

All results produced by forecasting module 114 are stored in ML storage106. In some embodiments, this includes the selected, trained model andall of the features and hyperparameters associated thereof, along withthe forecast results.

Since ML storage 106 is updated dynamically with external data and userdata, the forecast results produced by forecasting module 114 can beevaluated by comparing with actual updated data. In some embodiments,the evaluation of forecasted results is performed by monitor module 112and used as part of the evaluating the forecasting step (block 804) inFIG. 8 .

FIG. 10 illustrates a flowchart of a forecast request evaluation by amonitor module 112 in accordance with one embodiment.

Monitor module 112 receives a request for a forecast at block 1002. Forexample, this can be a request to forecast the demand of a certainclient product at a particular location over a specific time period. Asan example, this may be a request to forecast the demand of a particularmake of a vacuum cleaner at a specific retail store in Kanata, Ontario,for a period of 13 weeks.

Monitor module 112 checks with the ML storage 106 to see if this isfirst time a forecasting request for this particular product andlocation is made, at block 1004. If it is, then monitor module 112 flagsthe request to undergo a full model selection process at block 1006,which is subsequently sent to forecasting module 114 (see FIG. 9 ).Embodiments of the full model selection process are described below.

If this is not the first time a forecasting request for this particularproduct and location is made, then monitor module 112 checks the MLstorage 106 to see if any new class of relevant signal data has beenadded since the last forecast request for the particular product andlocation, at block 1008. If the answer is yes, then monitor module 112flags the request to undergo a full model selection process at block1006, which is subsequently sent to forecasting module 114 (see FIG. 9).

As an example, in the intervening period between the first request andthe subsequent request, ML storage 106 may have received weather datathat includes a humidity index relevant to the location of the request,which was not present in the data used for the initial forecast. Thehumidity index is a new class of signal data that can be used in themachine learning forecasting of the particular product at the particularlocation. Note that if new humidity data has been received during theintervening period, but the new humidity data has no impact on thelocation of interest, then it is not considered as being relevant. Forexample, if ML storage 106 receives the humidity index for Washington,D.C., but not for Kanata ON (where the forecast is requested), then thisis not considered as a relevant new class of signal data.

If the answer at block 1008 is no, monitor module 112 checks to see ifthe amount of new relevant data stored in ML storage 106 since the lastforecast, is above a certain threshold at block 1010. If yes, thenmonitor module 112 flags the request to undergo a full model selectionprocess at ML storage 106, which is subsequently sent to forecastingmodule 114 (see FIG. 9 ). Again, a comparison is made with regards todata that is relevant to the forecasting request in question.

As an example, ML storage 106 receives user data and external data on anon-going basis. During an intervening period between forecast requests,the amount of new signal data accumulates, and can be quantified suchthat the amount of new data is a percentage of the amount of data usedfor the previous forecast. In some embodiments, the minimum thresholdmay be 15% to 30%, or 15% or 20% or 25% or 30%, of the amount of dataused for the previous forecast.

If the answer at block 1010 is no, monitor module 112 proceeds to block1012 to evaluate the performance of the machine learning model used inthe previous forecast. With reference to FIG. 9 , once the forecastingmodule 114 provides a forecast, the forecast is stored in the ML storage106. Monitor module 112 evaluates the forecast on an ongoing basis bycomparing the forecasted values with the actual values as the latter areuploaded to ML storage 106 on an ongoing basis. Evaluation methods knownin the art may be used to evaluate the accuracy of the forecastedvalues, and a criterion may be selected to determine whether or not theforecast remains viable. In some embodiments, the evaluation method canbe selected from mean absolute percentage error (MAPE); mean absolutescaled error (MASE), mean absolute error (MAE), and Weighted MeanAbsolute Percentage Error (WMAPE). If the forecast is not deemed viable,then monitor module 112 flags the request to undergo a full modelselection process at block 1006, which is subsequently sent toforecasting module 114 (see FIG. 9 ).

If the answer at block 1012 is no, monitor module 112 proceeds to block1012 to evaluate the time interval between the previous and currentforecasting requests at block 1014. If the time interval is beyond acertain threshold, then monitor module 112 flags the request to have thecurrent model retrained at block 1010, which is subsequently sent toforecasting module 114 (see FIG. 9 ). The time interval threshold may beany value. In some embodiments, the time threshold value may be a fewmonths or more. In some embodiments, the time threshold may be 2 to 8months, or 3 to 6 months, or 2 months, or 3 months, or 6 months.Retraining of the selected model may comprise retraining on the initialdata set along with the data that has accumulated in the interveningtime period.

If the time threshold is surpassed, all of the signal data gathered fromthe user and external data sources during the intervening period betweensuccessive forecasting requests is used to retrain the current model.Embodiments of retraining are described below.

If the time threshold is not surpassed, monitor module 112 proceeds toinstruct forecasting module 114 to forecast using the current model atblock 1018, without any retraining.

FIG. 11 illustrates a flow chart 1100 of a forecasting module 114 inaccordance with one embodiment.

Forecasting module 114 receives instructions from monitor module 112, asshown in FIG. 9 , to either select a model (block 902), train/retrain(block 904), or forecast (block 906). In FIG. 11 , block series 1118describes a flowchart of the model selection process 1102 in anembodiment; block series 1120 describes a flowchart of the trainingprocess 1112 in an embodiment, and block 1116 refers to the forecastingof the trained ML model.

When forecasting module 114 is instructed to conduct a model selectionprocess 1102 by monitor module 112, a first step is to engineer featuresof signal data (retrieved from ML storage 106) at step block 1104.Features engineering may be automated in the sense that the system cangenerate features more amenable to machine learning without having auser define one or more transformations of the features engineeringprocess. In some embodiments, selection of transformations may beachieved automatically through methods like cross-validation,dimensionality reduction and independence criteria tests.

The timeframe of signals data used at block 1104 varies, depending onwhere in flowchart 1000 (of FIG. 10 ) the model selection processoccurs. For example, if the model selection process is selected at block1006 (i.e. performing a forecast for the first time), then all of therelevant signal data (from the user and external data sources) collectedup to the time of receiving the first forecast request, is used. Forexample, if the model selection process is selected at block 1006 (i.e.performing a subsequent forecast), then all of the relevant signal data(from the user and external data sources) collected up to the time ofreceiving the subsequent request, is used.

Once the features engineering data is prepared, it is used to train,validate and optionally test. In some embodiments, the engineered datais divided chronologically into two portions: a training portion and avalidation portion. Forecasting module 114 obtains access to a pluralityof ML models in ML storage 106; each of the ML models is trained on thetraining portion at block 1106. Each trained ML model then provides aprediction for the time period corresponding to the validation portionat block 1108. Each forecast is evaluated for accuracy, and the mostaccurate model is selected for further forecasting at block 1110.

In some embodiments, the training portion may be between 50%-80% of thefull engineered dataset; the validation portion may be between 20%-50%%,such that the overall total is 100%. In some embodiments, thetraining/validation portions may be 60%/40%%, or 80%/20% or 70%/30%.Different validation schemes, known in the art, may be used. Examplesinclude nested cross-validation comprising an outer loop and inner loop(for tuning hyperparameters). As an example, validation may be performedover many windows of the training data, rather than one single window.

Retraining of a selected ML model is described in block series 1120, inaccordance with one embodiment. A selected ML model is first retrainedon an expanded dataset at block 1114. Details of the training/retrainingvary slightly, depending on where in the overall process of FIG. 10 theselected model is being trained—within a model selection process (i.e.in block 1006); or within a retraining process alone (i.e. Block 1016).

Where retraining of the selected ML model is part of a model selectionprocess 1102, the selected ML model is retrained on an expandedengineered data set comprising data corresponding to the training andvalidation portions of the dataset (at block 1114).

Where retraining of the selected ML model occurs without model selectionprocess 1102 (i.e. retraining only), the selected ML model is retrainedon an expanded engineered data set comprising data corresponding to thetraining and validation portions of the dataset (at block 1114).

FIG. 12 illustrates a flow chart 1200 of a forecasting module 114 inaccordance with one embodiment.

Forecasting module 114 receives instructions from monitor module 112, asshown in FIG. 9 , to either select a model (block 902), train/retrain(block 904), or forecast (block 906). In FIG. 12 , block series 1222describes a flowchart of the model selection process 1202 in anembodiment; block series 1224 describes a flowchart of the trainingprocess 1212 in an embodiment, and block 1220 refers to the forecastingof the trained ML model.

When forecasting module 114 is instructed to conduct a model selectionprocess 1202 by monitor module 112, a first step is to engineer featuresof signal data (retrieved from ML storage 106) at step block 1204.Features engineering may be automated in the sense that the system cangenerate features more amenable to machine learning without having auser define one or more transformations of the features engineeringprocess. In some embodiments, selection of transformations may beachieved automatically through methods like cross-validation,dimensionality reduction and independence criteria tests.

The timeframe of signals data used at block 1204 varies, depending onwhere in flowchart 1000 (of FIG. 10 ) the model selection processoccurs. For example, if the model selection process is selected at block1006 (i.e. performing a forecast for the first time), then all of therelevant signal data (from the user and external data sources) collectedup to the time of receiving the first forecast request, is used. Forexample, if the model selection process is selected at block 1006 or MLstorage 106 (i.e. performing a subsequent forecast), then all of therelevant signal data (from the user and external data sources) collectedup to the time of receiving the subsequent request, is used.

Once the features engineering data is prepared, it is used to train,validate and test. In some embodiments, the engineered data is dividedchronologically into three portions: a training portion, a validationportion and a test portion. Forecasting module 114 obtains access to avariety of ML models in ML storage 106; each of the ML models is trainedon the training portion at block 1206. Each trained ML model thenprovides a prediction for the time period corresponding to thevalidation portion at block 1208. Each forecast is evaluated foraccuracy, and the most accurate model is selected for furtherforecasting at block 1210.

In some embodiments, the training portion may be between 50%-80% of thefull engineered dataset; the validation portion may be between 10%-25%,and the testing portion between 5%-25%, such that the overall total is100%. In some embodiments, the training/validation/testing portions are60%/20%/20%. In some embodiments, the amount of data in the testingportion may be too large for timely execution, in which case, anabsolute time frame of data is chosen. As an example, this may bebetween the last 10-20 weeks of data, or about the last 15 weeks ofdata, while increasing the size of the training and/or validationportion. Other validation schemes, known in the art, may be used.Examples include nested cross-validation comprising an outer loop andinner loop (for tuning hyperparameters). As an example, validation maybe performed over many windows of the training data, rather than onesingle window.

Retraining of a selected ML model is described in block series 1224, inaccordance with one embodiment. A selected ML model is first retrainedon an expanded dataset at block 1214; it then makes a forecastcorresponding to the period of a testing portion at block 1216, and itsaccuracy is evaluated, based on its performance in the testing portion,at block 1218. Details of the training/retraining vary slightly,depending on where in the overall process of FIG. 10 , the selectedmodel is being trained—within a model selection process (i.e. in block1006, block 1006, ML storage 106 or 618); or within a retraining processalone (i.e. Block 1006).

Where retraining of the selected ML model is part of a model selectionprocess 1202, the selected ML model is retrained on an expandedengineered data set comprising data corresponding to the training andvalidation portions of the dataset (at block 1214). It then forecastspredictions corresponding to the time period of the testing portion atblock 1216, and its accuracy evaluated at block 1218.

Where retraining of the selected ML model occurs without model selectionprocess 1202 (i.e. retraining only), the selected ML model is retrainedon an expanded engineered data set comprising data corresponding to thetraining and validation portions of the dataset (at block 1214). It thenforecasts predictions corresponding to the time period of the testingportion at block 1216, and its accuracy evaluated at block 1218.

Accuracy evaluation of the selected ML model, at block 1218, may beperformed by a number of techniques know in the art. For example, errormeasures such as Mean Absolut Error (MAE) or Mean Absolute Scaled Error(MASE) can be used as an evaluation of accuracy. Accuracy evaluationallows a user to have an idea of demand predictability of a particularproduct at a particular location for a specified time period.

EXAMPLE

A client requests a 13-week forecast for demand of a particular productat its only store in Kanata, Ontario. The user provides historical datarelated to the sales of the product at its Kanata location, in the formof point-of-sales data, inventory, promotional information, and the liketo ingestion module 302 (of FIG. 1 ) which pre-processes the data andstores it in ML storage 106. In addition, the user provides data relatedto the sales of the product at the Kanata location on an ongoing basis,such that the ML storage 106 is updated on an ongoing basis.

External data related to the product sales in Kanata is obtained byexternal data module 110 from external data source(s) 108. Such externaldata includes historical weather data for Kanata and the surroundingregion, holidays and events throughout the year for Kanata, marketindexes for the province of Ontario (in which Kanata is located) andCanada, along with other financial data related to Kanata. The externaldata is also pre-processed and stored in ML storage 106. As with userdata, the external data is updated on an ongoing basis.

The monitor module 112 receives the forecast request from request module116 and determines whether or not this request is the first request forthe particular product at its Kanata store. If it is the first requestof its kind, monitor module 112 flags the forecast request for a modelselection process by forecasting module 114. On the other hand, it thisis a subsequent forecast request, monitor module 112 proceeds throughthe flowchart shown in FIG. 10 , and flags the forecast request to theforecasting module 114 if necessary, or simply instructs the forecastingmodule 114 to forecast the demand for the next 13 weeks.

Where the forecasting request is the first of its kind, forecastingmodule 114 conducts a model selection process in which all historicaldata relevant to the particular product at the Kanata store is queriedfrom the ML storage 106. Forecasting module 114 then generates features(for both external data and user-supplied data) and merged with demanddata (the target for the forecast—data already supplied by user). Atraining and validation set are created from the whole historical setand various methods are applied to find the best configuration offeatures and models, where each configuration is compared and the bestone according to a metric of choice (e.g. MAE, MASE, etc.) is picked.This configuration (e.g. features and hyperparameters) is saved to thedatastore and the selected model is trained on a larger training set andits performance is measured on a testing set that corresponds to themost recent data acquired about the product sales at the Kanata store.The ML model and its performance are also stored in ML storage 106.

Then a forecast is computed with the trained model for a period of 13weeks ahead. In addition, explanations of each point of the forecast arecomputed according to the features used by the model and what theyentailed for each prediction.

FIG. 13 illustrates a machine learning forecast 1300 in accordance withone embodiment. In FIG. 13 , a request is made for the first time. Thesales of the product at a particular store in Kanata is shown on theY-axis, against a calendar date (from 2016-2018) on the X-axis.

Actual sales are shown (by a solid line) from about September 2016 toAugust 2018, with sales peaking in 2017 around the month of August, andin 2018 around July-August. Thus, the historical data set is fromSeptember 2016 to August 2018, a period of 2 years. The historical dataconsisted of daily sales at one store for the one product, whileexternal data consisted of holidays (i.e. no weather information, nomarket indexes).

The total period of initial training and validation is from September2016 to Jan. 15, 2018, after which, testing takes place (as indicated byblack dot 1302). A number of ML models, such as gradient-boosted trees,ensemble of trees and support vector regression, were used during theinitial training set. A gradient-boosted tree model, Light GBM, wasselected during validation, and retrained on the dataset from September2016 to Jan. 15, 2018. In this example, all the data, except for thelast 20%, was used for training the selected model. In some embodiments,the testing dataset may be the smaller of the dataset of the period ofthe last 10-20 weeks and the last 20% of the entire dataset. In someembodiments, where the historical data set spans 1 year (52 weeks), thetraining/validation period can be 40-42 weeks, with remaining 10-12weeks used for testing the selected model. In some embodiments, a nestedvalidation scheme can be used. The best ML model may be selectedaccording to a configuration set by the user, or any standard criteriasuch as MASE, MAE, WMAPE (Weighted Mean Absolute Percentage Error), etc.

In FIG. 13 , results are shown for the test period of Jan. 15, 2018-Aug.31, 2018. The forecasted ML results are shown as “predicted sale” (linewith solid circles) in the legend table 1304, while the actual sales arelabeled as ‘weekly sales” (solid line). For comparison, a seasonalforecast (line with solid triangles) is also shown for the test period.The seasonal forecast is implemented by simply copying the weekly salesthat happened one year prior. In this example, the season forecast forthe weeks of Feb. 20, 2018 to Aug. 31, 2018 is identical to the weeklysales during the weeks of Feb. 20, 2017 to Aug. 31, 2018.

Accuracy of the ML model in FIG. 13 was evaluated relative to theaccuracy of the seasonal forecast, using MASE. With the seasonalforecast accuracy normalized to a MASE value of 1.0, the ML model wasfound to have a MASE value of 0.81; i.e. the ML model was more accurateat predicting weekly sales than the seasonal forecast method.

In addition, FIG. 13 provides an interval of sales between the predicted10th and 90th percentiles for the ML model. This interval was producedusing quantile regression. In some embodiments, a quintile regressionapproach may not be possible for all ML models. That is, other ML modelsmay use a different scheme (i.e. not quantile regression) to providesuch an interval.

Machine learning models are also referred to herein as machine learningforecast models.

Demand sensing is commonly relied upon by retailers and manufacturers toensure adequate supply of a product is in their stores and inventory tomeet customer demand. A simple flowchart for an exemplary Method 1400for forecasting using a machine learning technique according to anembodiment is shown in FIG. 14 .

Method 1400 will be described as carried out by System 1502 shown inFIG. 15 . Alternatively, Method 1400 may be carried out by anothersystem, a combination of other systems, subsystems, devices or othersuitable means provided the operations described herein are performed.Method 1400 may be automated, semi-automated and some blocks thereof maybe manually performed.

In FIG. 15 System 1502 is shown in an exemplary Environment 1500 withwhich some embodiments may operate. System 1502 includes a Memory Store1504 and a Processing Resource 1506. Processing Resource 1506 mayinclude one or more processors and/or controllers, which may take theform of a general or a special purpose processor(s) or controller(s). Inexemplary implementations, Processing Resource 1506 may be, or include,microprocessors, microcontrollers, application specific integratedcircuits, digital signal processors, and/or other data processingdevices. Processing Resource 1506 may be a single device or distributedover a network.

Memory Store 1504 may be or include one or more non-transitorycomputer-readable storage media, such as optical, magnetic, organic, orflash memory, among other data storage devices and may take any form ofcomputer readable storage media. Remote Data store 1510 may be a singledevice or may be distributed over a network.

Processing Resource 1506 and Memory Store 1504 may be communicativelycoupled by a system communication bus, a wired network, a wirelessnetwork, or other connection mechanism and arranged to carry out variousoperations described herein. Optionally, two or more of these componentsmay be integrated together in whole or in part.

System 1502 is communicatively coupled to a communication Network 1508as shown by arrow 1516. Communication Network 1508 may include one ormore computing systems and may be any suitable combination of networksor portions thereof to facilitate communication between networkcomponents. Some examples of networks include, Wide Area Networks(WANs), Local Area Networks (LANs), Wireless Wide Area Networks (WWANs),data networks, cellular networks, voice networks, among other networks,which may be wired and/or wireless. Communication Network 1508 mayoperate according to one or more communication protocols, such as,General Packet Radio Service (GPRS), Universal Mobile TelecommunicationsService (UMTS), Global System for Mobile (GSM), Enhanced Data Rates forGSM Evolution (EDGE), Long Term Evolution (LTE), CDMA (Code-divisionMultiple Access) (CDMA), WCDMA (Wide Code-division Multiple Access),(High Speed Packet Access (HSPA), Evolved HSPA (HSPA+), Low-power WAN(LPWAN), Wi-Fi, Bluetooth, Ethernet, Hypertext Transfer Protocol Secure(HTTP/S), Transmission Control Protocol/Internet Protocol (TCP/IP), andConstrained Application Protocol/Datagram Transport Layer Security(CoAP/DTLS), or other suitable protocol. Communication Network 1508 maytake other forms as well.

Also shown in FIG. 15 is a remote Data store 1510, remote Client server1512 and remote third Party server 1514, each communicatively coupled tocommunication Network 1508, as shown by arrow 1518, arrow 1520, andarrow 1522, respectively. For example, System 1502 can communicate withData store 1510, Client server 1512 and third Party server 1514, viacommunication Network 1508. Furthermore, Data store 1510 can communicatewith Client server 1512 and third Party server 1514, via communicationNetwork 1508.

In a first example, a retailer owns 1400 stores located at variouslocations. The retailer, also referred to herein as the user, wishes toreceive, for a 4-week (28 day) forecast window, a forecast for the dailydemand of a first product for each of the 1400 store locations.Alternatively, and/or optionally, the forecast may be for a differentproduct, a unique store(s), a different forecast window, etc.

At block 1402, Method 1400 includes collecting historical data during afirst time interval. For example, the retailer transmits sales data ofthe first product on a daily basis corresponding to each store locationfrom Client server 1512 to Data store 1510. The sales data is collectedand stored daily in Data store 1510 over a first time interval forfuture use. In this example, the first time interval includes 1 year. Inpractice, however, sales data may be collected over any time interval,e.g., weeks, months, years, etc. In this example, daily weather datacorresponding to each day of that 1 year period is also collected. Forinstance, daily weather data from third Party server 214, such as aserver storing weather conditions, is fetched by Processing Resource1506 and stored in Data store 1510 for future use. Sales datatransmitted by the retailer and the weather data fetched by ProcessingResource 1506 over the 1 year period is referred to as historical data.Alternatively, other third party signals may include the time of year,local holidays, events, market indexes at the location, amongst others.

Next, at block 1404, Method 1400 includes generating feature dataindicative of features used for training a machine learning algorithm.For example, Processing Resource 1506 may massage, process and/ortransform historical data for generating feature data used to train amachine learning algorithm for creating a forecast model.

Next, at block 1406, Method 1400 includes training a machine learningalgorithm using the feature data. For example, Processing Resource 1506,trains a machine learning algorithm, such as, a tree-based machinelearning algorithm, using the generated feature data for forming aforecast model. Optionally, at block 1406, hyper-parameters of theforecast model may be tuned to improve accuracy of the model.

Next, at block 1410, Method 1400 includes providing forecast data. Forexample, using the forecasting model, Processing Resource 1506 generatesforecast data indicating the daily demand for the first product for eachstore of the 1400 stores, over a 4-week period.

Next, at block 1412, Method 1400 includes transmitting forecast data tothe user. For example, Processing Resource 1506 transmits the forecastdata to Client server 1512 via communication Network 1508.

Next, at block 1414, Method 1400 includes collecting new data over asecond period of time. For example, the retailer transmits sales datafor the first product on a daily basis corresponding to each store fromClient server 1512 to Data store 1510. The daily sales data is collectedover the second time interval, for instance, 1 week, and stored in Datastore 1510 for future use. In practice, however, sales data may becollected over any time interval, e.g., days, weeks, months, etc. Inthis example, daily weather data corresponding to each day of that 1week period is also collected. For instance, daily weather data fromthird Party server 1514, such as a server storing weather conditions, isfetched by Processing Resource 1506 and stored in Data store 1510 forfuture use. Sales data transmitted by the retailer and the weather datafetched by Processing Resource 1506 over the 1 week period is referredto as new data.

At block 1416, the new data is added to the historical data. Forexample, the new data is added to the historical data stored in Datastore 1510.

Finally, Method 1400 returns to block 1404 wherein feature data isgenerated. Feature data is based on the new data and is used fortraining a machine learning algorithm for forming another forecastmodel.

In practice, Method 1400 may be a portion of a machine learningpipeline. In such cases, the size of historical data and/new data may belarge, e.g., terabytes, petabytes, etc., causing implementation ofMethod 1400 in a machine learning pipeline to be expensive in terms oftime, cost and resources. In the example provided above, although salesdata is collected on a daily basis, feature generation, training amachine learning algorithm, and providing a forecast occurs once every 7days. In some instances, however, a retailer/user may request to receivean updated forecast on a more frequent basis based the most recent salesdata available. To accommodate the user's request, the machine learningpipeline would need to be run more frequently, which is expensive asdescribed above, and/or, the pipeline may need re-structuring, which maynot be feasible.

According to an embodiment there is a Method 1600 for adjusting forecastdata provided by a machine learning model based on new sales data andforecast data. Shown in FIG. 16 is a flow diagram of Method 1600. Method1600 is described as being carried out by System 1502.

Method 1600 includes blocks 1602, 1604, 1606, and 1608, sequentially,and is included in Method 1400 located between block 1412 and block 1414of Method 1400, as indicated by bracket 1418.

Continuing with the same example as described herein above withreference to Method 1400, the retailer wishes to receive a dailyforecast for each first product corresponding to each of the 1400 storelocations.

At block 1602, Method 1600 includes collecting new sales data during asecond time interval. For example, the retailer transmits sales data forDay 1 of the 28 day (4-week) forecast window. The sales data includessales data of the first product corresponding to each store. The salesdata is transmitted from Client server 1512 to Data store 1510 andstored in Data store 1510 for future use.

Next, at block 1604, Method 1600 includes determining an error based onforecast data provided at block 1410 and new sales (i.e., actual sales)data collected during the second time interval. For example, ProcessingResource 1506 determines an error in the form of an average percentageof error across all stores for the first product between the forecastdata and the new sales (i.e., actual sales) data for Day 1.

For instance, for a first store of the 1400 stores, Processing Resource1506 determines the percentage error between the forecasted demand ofthe first product corresponding to that store on Day 1 and the new sales(i.e., actual sales) data collected on Day 1. Forecast data for Day 1indicates product demand for the first product corresponding to thefirst store to be 10 units. The new sales (i.e., actual sales) datacollected for the first product and corresponding to the first storeindicates 9 units were actually sold. Processing Resource 1506determines there is a percentage error of (10-9)/1400=0.10 or 10% errorbetween the forecast data and the new sales (i.e., actual sales) datafor the first store location during the second time interval.

Next, for each of the remaining 99 store locations, Processing Resource1506 determines the percentage error between the forecasted demand ofthe first product and the new sales (i.e., actual sales) data for Day 1in a similar manner as described above.

Then Processing Resource 1506 averages the percentage error of allstores. For example, Processing Resource 1506 determines the averagepercentage error across all stores is 0.02%.

Referring now to FIG. 17 , shown in table 1700 is exemplary forecastdata 1704 for the first product of a first store over a 4-week window(28 days) provided by the forecast model. For example, as shown incolumn 1702, on Day 1 10 units of the first product was forecast to besold at the first store. In a second example, as shown in column 1702,on Day 11 18 units of the first product was forecast to be sold at thefirst store.

Next at block 1606, Method 1600 includes removing the error from theforecast data for forming adjusted forecast data. For example,Processing Resource 1506 removes the average percentage error across allstores, 0.02%, from the forecast data for each of the 28 day forecastwindow, for forming adjusted forecast data. Exemplary adjusted forecastdata is shown in column 1706 in table 1700. Now referring to FIG. 18 ,shown is a graphical representation of forecast data 1802 and adjustedforecast data 1804.

Finally, at block 1608, Method 1600 includes transmitting the adjustedforecast data. For example, Processing Resource 1506 transmits theadjusted forecast data to Client server 1512 via communication Network1508.

Optionally, Method 1600 returns to block 1602 as indicated by arrow1610.

At block 1602, Method 1600 includes collecting new sales data during athird time interval. For example, the retailer transmits sales data forDay 2 of the 28 day (4-week) forecast window. The sales data includessales data of the first product corresponding to each store. The salesdata is transmitted from Client server 1512 to Data store 1510 andstored in Data store 1510 for future use.

Next, at block 1604, Method 1600 includes determining an error based onforecast data provided at block 1410 and new sales (i.e., actual sales)data collected during the third time interval. For example, ProcessingResource 1506 determines an error in the form of an average percentageof error across all stores for the first product between the forecastdata and the new sales (i.e., actual sales) data for Day 2.

For example, Processing Resource 1506 determines the average percentageerror across all stores in a similar manner as described above to be0.77%.

Next, at block 1606, Method 1600 includes removing the error from theforecast data for forming adjusted forecast data. For example,Processing Resource 1506 removes the average percentage error of allstores, 0.77%, from the forecast data for each of the 28 day forecastwindow, for forming second adjusted forecast data. Exemplary secondadjusted forecast data is shown in column 1708 in table 1700. Nowreferring to FIG. 18 , shown is a graphical representation of secondadjusted forecast data 1806.

Finally, at block 1608, Method 1600 includes transmitting the adjustedforecast data. For example, Processing Resource 1506 transmits thesecond adjusted forecast data to Client server 1512 via communicationNetwork 1508.

In the present example, Method 1400 repeats blocks 1602, 1604, 1606, and1608, in a similar manner as described above for days 3, 4, 5, and 6.

Aspects of the present disclosure may be embodied as a system, method orcomputer program product. Accordingly, aspects of the present disclosuremay take the form of an entirely hardware embodiment, an entirelysoftware embodiment (including firmware, resident software, micro-code,etc.) or an embodiment combining software and hardware aspects that mayall generally be referred to herein as a “circuit,” “module” or“system.” Furthermore, aspects of the present disclosure may take theform of a computer program product embodied in one or more computerreadable storage media having computer readable program code embodiedthereon.

Many of the functional units described in this specification may belabeled as modules, in order to emphasize their implementationindependence. For example, a module may be implemented as a hardwarecircuit comprising custom VLSI circuits or gate arrays, off-the-shelfsemiconductors such as logic chips, transistors, or other discretecomponents. A module may also be implemented in programmable hardwaredevices such as field programmable gate arrays, programmable arraylogic, programmable logic devices or the like.

Modules may also be implemented in software for execution by varioustypes of processors. An identified module of executable code may, forinstance, comprise one or more physical or logical blocks of computerinstructions which may, for instance, be organized as an object,procedure, or function. Nevertheless, the executables of an identifiedmodule need not be physically located together, but may comprisedisparate instructions stored in different locations which, when joinedlogically together, comprise the module and achieve the stated purposefor the module.

Indeed, a module of executable code may be a single instruction, or manyinstructions, and may even be distributed over several different codesegments, among different programs, and across several memory devices.Similarly, operational data may be identified and illustrated hereinwithin modules, and may be embodied in any suitable form and organizedwithin any suitable type of data structure. The operational data may becollected as a single data set, or may be distributed over differentlocations including over different storage devices, and may exist, atleast partially, merely as electronic signals on a system or network.Where a module or portions of a module are implemented in software, thesoftware portions are stored on one or more computer readable storagemedia.

Any combination of one or more computer readable storage media may beutilized. A computer readable storage medium may be, for example, butnot limited to, an electronic, magnetic, optical, electromagnetic,infrared, or semiconductor system, apparatus, or device, or any suitablecombination of the foregoing.

More specific examples (a non-exhaustive list) of the computer readablestorage medium can include the following: a portable computer diskette,a hard disk, a random access memory (RAM), a read-only memory (ROM), anerasable programmable read-only memory (EPROM or Flash memory), aportable compact disc read-only memory (CD-ROM), a digital versatiledisc (DVD), a Blu-ray disc, an optical storage device, a magnetic tape,a Bernoulli drive, a magnetic disk, a magnetic storage device, a punchcard, integrated circuits, other digital processing apparatus memorydevices, or any suitable combination of the foregoing, but would notinclude propagating signals. In the context of this document, a computerreadable storage medium may be any tangible medium that can contain, orstore a program for use by or in connection with an instructionexecution system, apparatus, or device.

Computer program code for carrying out operations for aspects of thepresent disclosure may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Python, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Reference throughout this specification to “one embodiment,” “anembodiment,” or similar language means that a particular feature,structure, or characteristic described in connection with the embodimentis included in at least one embodiment of the present disclosure. Thus,appearances of the phrases “in one embodiment,” “in an embodiment,” andsimilar language throughout this specification may, but do notnecessarily, all refer to the same embodiment, but mean “one or more butnot all embodiments” unless expressly specified otherwise. The terms“including,” “comprising,” “having,” and variations thereof mean“including but not limited to” unless expressly specified otherwise. Anenumerated listing of items does not imply that any or all of the itemsare mutually exclusive and/or mutually inclusive, unless expresslyspecified otherwise. The terms “a,” “an,” and “the” also refer to “oneor more” unless expressly specified otherwise.

Furthermore, the described features, structures, or characteristics ofthe disclosure may be combined in any suitable manner in one or moreembodiments. In the following description, numerous specific details areprovided, such as examples of programming, software modules, userselections, network transactions, database queries, database structures,hardware modules, hardware circuits, hardware chips, etc., to provide athorough understanding of embodiments of the disclosure. However, thedisclosure may be practiced without one or more of the specific details,or with other methods, components, materials, and so forth. In otherinstances, well-known structures, materials, or operations are not shownor described in detail to avoid obscuring aspects of the disclosure.

Aspects of the present disclosure are described below with reference toschematic flowchart diagrams and/or schematic block diagrams of methods,apparatuses, systems, and computer program products according toembodiments of the disclosure. It will be understood that each block ofthe schematic flowchart diagrams and/or schematic block diagrams, andcombinations of blocks in the schematic flowchart diagrams and/orschematic block diagrams, can be implemented by computer programinstructions. These computer program instructions may be provided to aprocessor(s) of a general purpose computer(s), special purpose computer,or other programmable data processing apparatus to produce a machine,such that the instructions, which execute via the processor of thecomputer or other programmable data processing apparatus, create meansfor implementing the functions/acts specified in the schematic flowchartdiagrams and/or schematic block diagrams block or blocks.

These computer program instructions may also be stored in a computerreadable storage medium that can direct a computer, other programmabledata processing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablestorage medium produce an article of manufacture including instructionswhich implement the function/act specified in the schematic flowchartdiagrams and/or schematic block diagrams block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The schematic flowchart diagrams and/or schematic block diagrams in theFigures illustrate the architecture, functionality, and operation ofpossible implementations of apparatuses, systems, methods and computerprogram products according to various embodiments of the presentdisclosure. In this regard, each block in the schematic flowchartdiagrams and/or schematic block diagrams may represent a module,segment, or portion of code, which comprises one or more executableinstructions for implementing the specified logical function(s).

It should also be noted that, in some alternative implementations, thefunctions noted in the block may occur out of the order noted in thefigures. For example, two blocks shown in succession may, in fact, beexecuted substantially concurrently, or the blocks may sometimes beexecuted in the reverse order, depending upon the functionalityinvolved. Other steps and methods may be conceived that are equivalentin function, logic, or effect to one or more blocks, or portionsthereof, of the illustrated figures.

Although various arrow types and line types may be employed in theflowchart and/or block diagrams, they are understood not to limit thescope of the corresponding embodiments. Indeed, some arrows or otherconnectors may be used to indicate only the logical flow of the depictedembodiment. For instance, an arrow may indicate a waiting or monitoringperiod of unspecified duration between enumerated steps of the depictedembodiment. It will also be noted that each block of the block diagramsand/or flowchart diagrams, and combinations of blocks in the blockdiagrams and/or flowchart diagrams, can be implemented by specialpurpose hardware-based systems that perform the specified functions oracts, or combinations of special purpose hardware and computerinstructions.

The description of elements in each figure may refer to elements ofproceeding figures. Like numbers refer to like elements in all figures,including alternate embodiments of like elements.

A computer program (which may also be referred to or described as asoftware application, code, a program, a script, software, a module or asoftware module) can be written in any form of programming language.This includes compiled or interpreted languages, or declarative orprocedural languages. A computer program can be deployed in many forms,including as a module, a subroutine, a stand-alone program, a component,or other unit suitable for use in a computing environment. A computerprogram can be deployed to be executed on one computer or can bedeployed on multiple computers that are located at one site ordistributed across multiple sites and interconnected by a communicationnetwork.

As used herein, a “software engine” or an “engine,” refers to a softwareimplemented system that provides an output that is different from theinput. An engine can be an encoded block of functionality, such as aplatform, a library, an object or a software development kit (“SDK”).Each engine can be implemented on any type of computing device thatincludes one or more processors and computer readable media.Furthermore, two or more of the engines may be implemented on the samecomputing device, or on different computing devices. Non-limitingexamples of a computing device include tablet computers, servers, laptopor desktop computers, music players, mobile phones, e-book readers,notebook computers, PDAs, smart phones, or other stationary or portabledevices.

The processes and logic flows described herein can be performed by oneor more programmable computers executing one or more computer programsto perform functions by operating on input data and generating output.The processes and logic flows can also be performed by, and apparatuscan also be implemented as, special purpose logic circuitry, e.g., anFPGA (field programmable gate array) or an ASIC (application specificintegrated circuit). For example, the processes and logic flows that canbe performed by an apparatus, can also be implemented as a graphicsprocessing unit (GPU).

Computers suitable for the execution of a computer program include, byway of example, general or special purpose microprocessors or both, orany other kind of central processing unit. Generally, a centralprocessing unit receives instructions and data from a read-only memoryor a random access memory or both. A computer can also include, or beoperatively coupled to receive data from, or transfer data to, or both,one or more mass storage devices for storing data, e.g., optical disks,magnetic, or magneto optical disks. It should be noted that a computerdoes not require these devices. Furthermore, a computer can be embeddedin another device. Non-limiting examples of the latter include a gameconsole, a mobile telephone a mobile audio player, a personal digitalassistant (PDA), a video player, a Global Positioning System (GPS)receiver, or a portable storage device. A non-limiting example of astorage device include a universal serial bus (USB) flash drive.

Computer readable media suitable for storing computer programinstructions and data include all forms of non-volatile memory, mediaand memory devices; non-limiting examples include magneto optical disks;semiconductor memory devices (e.g., EPROM, EEPROM, and flash memorydevices); CD ROM disks; magnetic disks (e.g., internal hard disks orremovable disks); and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described herein can be implemented on a computer having adisplay device for displaying information to the user and input devicesby which the user can provide input to the computer (for example, akeyboard, a pointing device such as a mouse or a trackball, etc.). Otherkinds of devices can be used to provide for interaction with a user.Feedback provided to the user can include sensory feedback (e.g., visualfeedback, auditory feedback, or tactile feedback). Input from the usercan be received in any form, including acoustic, speech, or tactileinput. Furthermore, there can be interaction between a user and acomputer by way of exchange of documents between the computer and adevice used by the user. As an example, a computer can send web pages toa web browser on a user's client device in response to requests receivedfrom the web browser.

Embodiments of the subject matter described in this specification may beimplemented in a computing system that includes: a front end component(e.g., a client computer having a graphical user interface or a Webbrowser through which a user can interact with an implementation of thesubject matter described herein); or a middleware component (e.g., anapplication server); or a back end component (e.g. a data server); orany combination of one or more such back end, middleware, or front endcomponents. The components of the system can be interconnected by anyform or medium of digital data communication, e.g., a communicationnetwork. Non-limiting examples of communication networks include a localarea network (“LAN”) and a wide area network (“WAN”).

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of what may beclaimed, but rather as descriptions of features that may be specific toparticular embodiments. Certain features that are described in thisspecification in the context of separate embodiments can also beimplemented in combination in a single embodiment. Conversely, variousfeatures that are described in the context of a single embodiment canalso be implemented in multiple embodiments separately or in anysuitable sub-combination. Moreover, although features may be describedabove as acting in certain combinations and even initially claimed assuch, one or more features from a claimed combination can in some casesbe excised from the combination, and the claimed combination may bedirected to a sub-combination or variation of a sub-combination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various system modulesand components in the embodiments described above should not beunderstood as requiring such separation in all embodiments, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

Although the algorithms described above including those with referenceto the foregoing flow charts have been described separately, it shouldbe understood that any two or more of the algorithms disclosed hereincan be combined in any combination. Any of the methods, modules,algorithms, implementations, or procedures described herein can includemachine-readable instructions for execution by: (a) a processor, (b) acontroller, and/or (c) any other suitable processing device. Anyalgorithm, software, or method disclosed herein can be embodied insoftware stored on a non-transitory tangible medium such as, forexample, a flash memory, a CD-ROM, a floppy disk, a hard drive, adigital versatile disk (DVD), or other memory devices, but persons ofordinary skill in the art will readily appreciate that the entirealgorithm and/or parts thereof could alternatively be executed by adevice other than a controller and/or embodied in firmware or dedicatedhardware in a well-known manner (e.g., it may be implemented by anapplication specific integrated circuit (ASIC), a programmable logicdevice (PLD), a field programmable logic device (FPLD), discrete logic,etc.). Further, although specific algorithms are described withreference to flowcharts depicted herein, persons of ordinary skill inthe art will readily appreciate that many other methods of implementingthe example machine readable instructions may alternatively be used. Forexample, the order of execution of the blocks may be changed, and/orsome of the blocks described may be changed, eliminated, or combined.

It should be noted that the algorithms illustrated and discussed hereinas having various modules which perform particular functions andinteract with one another. It should be understood that these modulesare merely segregated based on their function for the sake ofdescription and represent computer hardware and/or executable softwarecode which is stored on a computer-readable medium for execution onappropriate computing hardware. The various functions of the differentmodules and units can be combined or segregated as hardware and/orsoftware stored on a non-transitory computer-readable medium as above asmodules in any manner and can be used separately or in combination.

Particular embodiments of the subject matter have been described. Otherembodiments are within the scope of the following claims. For example,the actions recited in the claims can be performed in a different orderand still achieve desirable results. As one example, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In certain implementations, multitasking and parallelprocessing may be advantageous.

What is claimed is:
 1. A computer-implemented method comprising: using afirst forecast model for providing first forecast data for a firstforecast window, the first forecast data for one or more storelocations; transmitting the first forecast data to a user; receivingfirst sales data for the one or more store locations for a second timeinterval, the second time interval subsequent a first time interval;determining an error in the first forecast data based on the firstforecast data and the first sales data; removing the error from thefirst forecast data for forming first adjusted forecast data, the firstadjusted forecast data for the one or more store locations; andtransmitting the first adjusted forecast data to the user.
 2. The methodof claim 1 including: collecting historical data for the first timeinterval; generating first feature data based on the historical data;and processing the first feature data by a processor for training amachine learning algorithm for forming the first forecast model.
 3. Themethod of claim 1 wherein determining the error in the first forecastdata based on the first forecast data and the first sales data includesdetermining the error between the first forecast data for each of theone or more store locations and the first sales data for each of the oneor more store locations corresponding thereto.
 4. The method of claim 1wherein determining the error in the first forecast data based on thefirst forecast data and the first sales data includes determining apercentage error between the first forecast data for each of the one ormore store locations and the first sales data for each of the one ormore store locations and an average percentage error thereof.
 5. Themethod of claim 4 wherein removing the error from the first forecastdata for forming the first adjusted forecast data includes removing theaverage percentage error from the first forecast data for each of theone or more store locations, the first adjusted forecast data includesfirst adjusted forecast data for each of the one or more storelocations.
 6. The method of claim 1 further including, collecting seconddata for a third time interval, the second data including at leasthistorical data and the first sales data; generating second feature databased on the second data; processing second feature data by a processorfor training a machine learning algorithm for forming a second forecastmodel; using the second forecast model for providing second forecastdata for a second forecast window, the second forecast data includingsecond forecast data for the one or more store locations; andtransmitting the second forecast data to the user.
 7. The method ofclaim 1 further includes, receiving second sales data, the second salesdata for the one or more store locations during a fourth time intervalsubsequent the second time interval; determining an error in the firstforecast data based on the first forecast data and the second salesdata; removing an error from the first forecast data for forming thirdforecast data; and transmitting the third forecast data to the user. 8.A non-transitory computer-readable storage medium, the computer-readablestorage medium including instructions that when executed by a computer,cause the computer to: use a first forecast model for providing firstforecast data for a first forecast window, the first forecast data forone or more store locations; transmit the first forecast data to a user;receive first sales data for the one or more store locations for asecond time interval, the second time interval subsequent a first timeinterval; determine an error in the first forecast data based on thefirst forecast data and the first sales data; remove the error from thefirst forecast data for forming first adjusted forecast data, the firstadjusted forecast data for the one or more store locations; and transmitthe first adjusted forecast data to the user.
 9. The non-transitorycomputer-readable storage medium of claim 8, wherein thecomputer-readable storage medium further includes instructions that whenexecuted by a computer, cause the computer to: collect historical datafor the first time interval; generate first feature data based on thehistorical data; and process the first feature data by a processor fortraining a machine learning algorithm for forming the first forecastmodel.
 10. The non-transitory computer-readable storage medium of claim8, wherein determining the error in the first forecast data based on thefirst forecast data and the first sales data includes determining theerror between the first forecast data for each of the one or more storelocations and the first sales data for each of the one or more storelocations corresponding thereto.
 11. The non-transitorycomputer-readable storage medium of claim 8, wherein determining theerror in the first forecast data based on the first forecast data andthe first sales data includes determining a percentage error between thefirst forecast data for each of the one or more store locations and thefirst sales data for each of the one or more store locations and anaverage percentage error thereof.
 12. The non-transitorycomputer-readable storage medium of claim 11, wherein removing the errorfrom the first forecast data for forming the first adjusted forecastdata includes removing the average percentage error from the firstforecast data for each of the one or more store locations, the firstadjusted forecast data includes first adjusted forecast data for each ofthe one or more store locations.
 13. The non-transitorycomputer-readable storage medium of claim 8, wherein thecomputer-readable storage medium further includes instructions that whenexecuted by a computer, cause the computer to: collect second data for athird time interval, the second data including at least historical dataand the first sales data; generate second feature data based on thesecond data; process second feature data by a processor for training amachine learning algorithm for forming a second forecast model; use thesecond forecast model for providing second forecast data for a secondforecast window, the second forecast data include second forecast datafor the one or more store locations; and transmit the second forecastdata to the user.
 14. The non-transitory computer-readable storagemedium of claim 8, wherein the computer-readable storage medium furtherincludes instructions that when executed by a computer, cause thecomputer to: receive second sales data, the second sales data for theone or more store locations during a fourth time interval subsequent thesecond time interval; determine an error in the first forecast databased on the first forecast data and the second sales data; remove anerror from the first forecast data for forming third forecast data; andtransmit the third forecast data to the user.
 15. A computing apparatuscomprising: a processor; and a memory storing instructions that, whenexecuted by the processor, configure the apparatus to: use a firstforecast model for providing first forecast data for a first forecastwindow, the first forecast data for one or more store locations;transmit the first forecast data to a user; receive first sales data forthe one or more store locations for a second time interval, the secondtime interval subsequent a first time interval; determine an error inthe first forecast data based on the first forecast data and the firstsales data; remove the error from the first forecast data for formingfirst adjusted forecast data, the first adjusted forecast data for theone or more store locations; and transmit the first adjusted forecastdata to the user.
 16. The computing apparatus of claim 15 furtherconfigured to: collect historical data for the first time interval;generate first feature data based on the historical data; and processthe first feature data by the processor for training a machine learningalgorithm for forming the first forecast model.
 17. The computingapparatus of claim 15 wherein determining the error in the firstforecast data based on the first forecast data and the first sales dataincludes determining the error between the first forecast data for eachof the one or more store locations and the first sales data for each ofthe one or more store locations corresponding thereto.
 18. The computingapparatus of claim 15 wherein determining the error in the firstforecast data based on the first forecast data and the first sales dataincludes determining a percentage error between the first forecast datafor each of the one or more store locations and the first sales data foreach of the one or more store locations and an average percentage errorthereof.
 19. The computing apparatus of claim 18 wherein removing theerror from the first forecast data for forming the first adjustedforecast data includes removing the average percentage error from thefirst forecast data for each of the one or more store locations, thefirst adjusted forecast data includes first adjusted forecast data foreach of the one or more store locations.
 20. The computing apparatus ofclaim 15 further configured to: collect second data for a third timeinterval, the second data including at least historical data and thefirst sales data; generate second feature data based on the second data;process second feature data by the processor for training a machinelearning algorithm for forming a second forecast model; use the secondforecast model for providing second forecast data for a second forecastwindow, the second forecast data include second forecast data for theone or more store locations; and transmit the second forecast data tothe user.
 21. The computing apparatus of claim 15 further configured to:receive second sales data, the second sales data for the one or morestore locations during a fourth time interval subsequent the second timeinterval; determine an error in the first forecast data based on thefirst forecast data and the second sales data; remove an error from thefirst forecast data for forming third forecast data; and transmit thethird forecast data to the user.